Differential Item Functioning By Language on the new PROMIS® Physical Functioning Items 2.0 for Adults
4. DIF Analyses with lorDIF
Constantin Yves Plessen
19 March, 2024
Overview
This document performs the DIF Analyses used in the section: Identification of Uniform DIF in Select PROMIS Items: Findings from Lordif Analysis.
DIF USA - Germany - Argentina
data_usa_ger_arg <- data_complete
## Select only PFM items
items_usa_ger_arg <- data_usa_ger_arg %>%
dplyr::select(starts_with("PF")) %>%
dplyr::select_if(~ !any(is.na(.))) # remove items with missing values
##Create language vector
language_usa_ger_arg <- dplyr::recode(data_usa_ger_arg$country,
usa = "1",
ger = "2",
arg = "3")Results
items <- as.data.frame(items_usa_ger_arg)
group <- language_usa_ger_arg
dif_result_nr2 <- lordif::lordif(items,
group,
criterion = "R2",
pseudo.R2 = "Nagelkerke",
R2.change = 0.021, # we chose this threshold to match the items identified from the multiverse
minCell = 5)EM cycles terminated after 500 iterations.
EM cycles terminated after 500 iterations.Call:
lordif::lordif(resp.data = items, group = group, criterion = "R2",
pseudo.R2 = "Nagelkerke", R2.change = 0.021, minCell = 5)
$criterion
[1] "R2"
$alpha
[1] 0.01
$pseudo.R2
[1] "Nagelkerke"
$R2.change
[1] 0.021
$beta.change
[1] 0.1
$maxIter
[1] 10
$minCell
[1] 5
$stats
item ncat chi12 chi13 chi23 beta12 pseudo12.McFadden pseudo13.McFadden
1 1 5 0.0000 0.0000 0.4918 0.0204 0.0046 0.0048
2 2 5 0.0510 0.1467 0.6534 0.0059 0.0006 0.0007
3 3 5 0.0000 0.0000 0.4376 0.0084 0.0038 0.0039
4 4 5 0.0000 0.0000 0.0249 0.0100 0.0059 0.0065
5 5 5 0.0000 0.0000 0.8470 0.0033 0.0027 0.0027
6 6 5 0.0000 0.0000 0.0009 0.0008 0.0043 0.0057
7 7 5 0.0000 0.0000 0.3087 0.0067 0.0047 0.0050
8 8 5 0.0001 0.0000 0.0096 0.0165 0.0017 0.0025
9 9 5 0.0000 0.0000 0.0635 0.0537 0.0109 0.0114
10 10 5 0.0000 0.0000 0.0000 0.0421 0.0090 0.0112
11 11 5 0.0000 0.0000 0.0003 0.1218 0.0371 0.0390
12 12 5 0.0000 0.0000 0.0738 0.0174 0.0025 0.0030
13 13 5 0.0000 0.0000 0.0275 0.0126 0.0057 0.0065
14 14 5 0.0000 0.0000 0.0001 0.0109 0.0068 0.0086
15 15 5 0.0000 0.0000 0.1576 0.0062 0.0111 0.0114
16 16 5 0.0000 0.0000 0.5247 0.0132 0.0085 0.0086
17 17 5 0.0000 0.0000 0.5206 0.0129 0.0026 0.0027
18 18 5 0.0000 0.0000 0.0291 0.0269 0.0125 0.0132
19 19 5 0.0000 0.0000 0.0073 0.0005 0.0045 0.0054
20 20 5 0.0000 0.0000 0.0432 0.0195 0.0021 0.0027
21 21 5 0.0000 0.0000 0.0000 0.0020 0.0039 0.0074
22 22 5 0.0001 0.0000 0.0092 0.0091 0.0017 0.0026
23 23 5 0.0000 0.0000 0.7745 0.1314 0.0413 0.0413
24 24 5 0.0000 0.0002 0.4865 0.0157 0.0021 0.0023
25 25 5 0.0000 0.0001 0.5778 0.0192 0.0027 0.0029
26 26 5 0.0000 0.0000 0.1521 0.0132 0.0063 0.0067
27 27 5 0.0604 0.0208 0.0508 0.0011 0.0007 0.0015
28 28 5 0.0000 0.0000 0.2396 0.0365 0.0075 0.0077
29 29 5 0.0000 0.0000 0.3032 0.0630 0.0102 0.0105
30 30 5 0.0000 0.0000 0.0027 0.0053 0.0104 0.0116
31 31 5 0.0000 0.0000 0.2165 0.0386 0.0092 0.0095
32 32 5 0.0000 0.0000 0.6523 0.1785 0.0509 0.0510
33 33 5 0.0000 0.0000 0.0791 0.0145 0.0028 0.0033
34 34 5 0.0000 0.0000 0.0026 0.0311 0.0132 0.0142
35 35 5 0.0000 0.0000 0.0003 0.0015 0.0051 0.0066
pseudo23.McFadden pseudo12.Nagelkerke pseudo13.Nagelkerke
1 0.0001 0.0051 0.0053
2 0.0001 0.0007 0.0008
3 0.0002 0.0046 0.0048
4 0.0006 0.0070 0.0077
5 0.0000 0.0031 0.0031
6 0.0014 0.0056 0.0074
7 0.0002 0.0049 0.0051
8 0.0008 0.0029 0.0043
9 0.0005 0.0129 0.0135
10 0.0022 0.0157 0.0195
11 0.0018 0.0507 0.0531
12 0.0005 0.0030 0.0035
13 0.0007 0.0069 0.0078
14 0.0018 0.0083 0.0104
15 0.0003 0.0130 0.0133
16 0.0001 0.0098 0.0099
17 0.0001 0.0034 0.0036
18 0.0007 0.0145 0.0153
19 0.0009 0.0058 0.0069
20 0.0006 0.0025 0.0033
21 0.0034 0.0047 0.0087
22 0.0008 0.0024 0.0035
23 0.0000 0.0636 0.0636
24 0.0001 0.0032 0.0035
25 0.0001 0.0040 0.0041
26 0.0003 0.0130 0.0137
27 0.0008 0.0009 0.0018
28 0.0003 0.0085 0.0088
29 0.0003 0.0143 0.0147
30 0.0012 0.0156 0.0173
31 0.0003 0.0108 0.0111
32 0.0001 0.0678 0.0679
33 0.0004 0.0031 0.0036
34 0.0010 0.0230 0.0248
35 0.0014 0.0079 0.0100
pseudo23.Nagelkerke pseudo12.CoxSnell pseudo13.CoxSnell pseudo23.CoxSnell
1 0.0001 0.0048 0.0050 0.0001
2 0.0001 0.0007 0.0008 0.0001
3 0.0002 0.0043 0.0045 0.0002
4 0.0008 0.0067 0.0074 0.0007
5 0.0000 0.0029 0.0030 0.0000
6 0.0018 0.0053 0.0069 0.0017
7 0.0002 0.0046 0.0048 0.0002
8 0.0014 0.0028 0.0041 0.0013
9 0.0006 0.0123 0.0128 0.0006
10 0.0038 0.0150 0.0186 0.0036
11 0.0024 0.0465 0.0487 0.0022
12 0.0005 0.0029 0.0034 0.0005
13 0.0009 0.0065 0.0073 0.0008
14 0.0021 0.0078 0.0098 0.0020
15 0.0004 0.0124 0.0128 0.0004
16 0.0002 0.0090 0.0091 0.0002
17 0.0002 0.0031 0.0033 0.0002
18 0.0008 0.0137 0.0144 0.0007
19 0.0011 0.0055 0.0066 0.0011
20 0.0008 0.0023 0.0031 0.0007
21 0.0040 0.0044 0.0082 0.0038
22 0.0012 0.0023 0.0034 0.0011
23 0.0001 0.0608 0.0609 0.0001
24 0.0002 0.0030 0.0032 0.0002
25 0.0002 0.0036 0.0038 0.0002
26 0.0007 0.0124 0.0131 0.0007
27 0.0009 0.0008 0.0016 0.0008
28 0.0003 0.0082 0.0084 0.0003
29 0.0004 0.0132 0.0135 0.0003
30 0.0017 0.0147 0.0163 0.0016
31 0.0003 0.0103 0.0106 0.0003
32 0.0001 0.0649 0.0650 0.0001
33 0.0005 0.0030 0.0034 0.0005
34 0.0018 0.0220 0.0237 0.0017
35 0.0022 0.0075 0.0096 0.0021
df12 df13 df23
1 2 4 2
2 2 4 2
3 2 4 2
4 2 4 2
5 2 4 2
6 2 4 2
7 2 4 2
8 2 4 2
9 2 4 2
10 2 4 2
11 2 4 2
12 2 4 2
13 2 4 2
14 2 4 2
15 2 4 2
16 2 4 2
17 2 4 2
18 2 4 2
19 2 4 2
20 2 4 2
21 2 4 2
22 2 4 2
23 2 4 2
24 2 4 2
25 2 4 2
26 2 4 2
27 2 4 2
28 2 4 2
29 2 4 2
30 2 4 2
31 2 4 2
32 2 4 2
33 2 4 2
34 2 4 2
35 2 4 2
$flag
[1] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE
[13] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE
[25] FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE TRUE FALSE
$flag.raw
[1] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE
[13] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE
[25] FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE TRUE FALSEPlot Results
Figure 1
Note. This graph shows smoothed histograms of the physical function
levels of study participants from the USA (black solid line), Germany
(orange dashed line), and Argentina (purple dotted line) as measured by
the PROMIS Physical Function scale (T-Scores) based on the baseline
model using the same parameters across countries. There is broad overlap
in the distributions, though individuals from the USA and Germany
demonstrated lower levels of Physical Function than individuals form
Argentina. There was a stronger ceiling effect for the USA distribution
at T-Score = 75 than for Germany or Argentina, which increased to T = 80
in the corrected model.
Figure 2
Fig. S5: Graphical display of the item PFM16 “Are you able to pass a 20-pound (10 kg) turkey or ham to other people at the table?” which shows uniform DIF with respect to country.
Note. The top-left graph presents the item characteristic curves (ICCs) for the item, contrasting the USA (black), Germany (orange dashed), and Argentina (purple dotted). The top-right graph illustrates the absolute differences in ICCs between the USA and the other two countries, highlighting that the major disparities for the USA with both Germany and Argentina occur at lower levels of physical function (theta). The bottom-left graph depicts the item response functions for each group, derived from demographic-specific item parameter estimates (slope and category threshold values), which are also annotated on the graph. Finally, the bottom-right graph displays the weighted absolute difference in ICCs (referencing the top-right graph) based on the score distribution of the focal group, the USA, showing a negligible impact.
Figure 3
Fig. S6: Graphical display of the item PFM33 “Are you able to walk across a balance beam?” which shows uniform DIF with respect to country.
Note. The upper-left graph illustrates the item characteristic curves (ICCs) for the item, comparing the USA (black), Germany (orange dashed), and Argentina (purple dotted). The upper-right graph portrays the absolute differences in ICCs between the USA and the other two countries, showing that the primary differences are at lower levels with Germany and at medium levels with Argentina in terms of physical function (theta). The lower-left graph presents the item response functions for each group, based on demographic-specific item parameter estimates (slope and category threshold values), which are also indicated on the graph. Finally, the lower-right graph demonstrates the weighted absolute difference in ICCs (from the upper-right graph) according to the score distribution of the focal group, the USA, suggesting a minimal impact.
Figure 4
Fig. S7: Graphical display of the item PFM46 “Are you able to pull a sled or a wagon with two children (total 100 lbs/50 kg) for 100 yards (100 m)?” which shows uniform DIF with respect to country.
Note. The top-left graph depicts the item characteristic curves (ICCs) for the item, showcasing comparisons between the USA (black), Germany (orange dashed), and Argentina (purple dotted). The top-right graph details the absolute differences in ICCs for the USA against Germany and Argentina, highlighting that the primary differences for Argentina compared to the other two nations occur at medium levels of physical function (theta). The bottom-left graph presents the item response functions for each group, derived from demographic-specific item parameter estimates (slope and category threshold values), which are also noted on the graph. Finally, the bottom-right graph displays the weighted absolute difference in ICCs (referenced in the top-right graph) based on the score distribution of the focal group, the USA, demonstrating a negligible impact.
Figure 5
Fig. S8: Graphical display of the item PFM51 “Are you able to swim laps for 30 minutes at a moderate pace?” which shows uniform DIF with respect to country.
Note. The upper-left graph displays the item characteristic curves (ICCs) for the item, contrasting the USA (black) with Germany (orange dashed) and Argentina (purple dotted). The upper-right graph demonstrates the absolute differences in ICCs for the USA compared to Germany and Argentina, revealing that the differences primarily occur at lower levels with Germany and at medium levels with Argentina in terms of physical function (theta). The lower-left graph presents the item response functions for each group, based on demographic-specific item parameter estimates (slope and category threshold values), also detailed on the graph. Finally, the lower-right graph illustrates the weighted absolute difference in ICCs (referenced in the upper-right graph) according to the score distribution of the focal group, the USA showing a minimal impact.
Figure 6
Fig. 5 Impact of DIF items on test characteristic curves
Note. These graphs show test characteristic curves (TCCs) for USA (black), Germany (orange dashed), and Argentina (purple dotted) using demographic–specific item parameter estimates. TCCs show the expected total scores for groups of items at each physical function level (theta). The graph on the left shows these curves for all of the items (both items with and without DIF), while the graph on the right shows these curves for the subset of these items found to have DIF. These curves suggest that at the overall test level there are some differences in the total expected score for individuals from all three countries. At the same level of ability (i.e., the same theta score on the x-axis), individuals from Germany obtained higher expected observed sum scores than while at the same level of ability individuals from Argentina had lower observed sum scores than the USA.
Figure 7
Fig. 6 Individual–level DIF impact.
Note. These graphs show the difference in score between using scores
that ignore DIF and those that account for DIF. The graph on the left
shows a box plot of these differences. The interquartile range,
representing the middle 50% of the differences (bound between the bottom
and top of the shaded box), range roughly from +0.02 to +0.03 with a
median of approximately +0.01. In the graph on the right the same
difference scores are plotted against the initial scores ignoring DIF
(“initial theta”), separately for individuals from the USA, Germany, and
Argentina. Guidelines are placed at 0.0 (solid line), i.e., no
difference, and the mean of the differences (dotted line). The overall
positive values for Germany indicate that accounting for DIF led to
slightly lower scores while the overall negative values for Argentina
indicate that accounting for DIF led to slightly higher scores. The
negative values for the USA to the left of this graph indicate that
accounting for DIF led to slightly higher scores for those with low
physical function, while the positive values to the right indicate
slightly lower scores for those with high physical function. The
positive values for Germany to the left of this graph indicate that
these effects were particularly pronounced in those with low physical
function.